شرح شغل

Analysis, design, development, and maintenance of data pipelines.Identify and design manual and repetitive processes and automate them with tools such as Airflow and Spark.Design the infrastructure required for optimal ETL from various sources of log files, SQL, and NoSQL databases.Microservice and event-sourcing-based design for data flow processing.Collaborate with various teams such as product, design, and data analysis to provide the necessary infrastructure.Implement all stacks with Docker-based tools.Experience working with stream processing tools such as KSQLDB, KSQL, and Kafka Stream.Required AbilitiesHaving in-depth knowledge of SQL and Python, Scala, or Java programming languages.Familiarity with Apache big data tools.Experience working with tools like Hadoop, Spark, and KafkaExperience working with SQL and NoSQL databases (PostgreSQL, MongoDB, etc.).Experience working with Kafka connector tools for SQL and NoSQL databases.Familiarity with automation tools such as Apache Airflow and Apache NiFi.Experience working with stream processing tools such as Kafka Streams. Having Teamwork skills and familiarity with Agile methodology.Ability to design and develop ML infrastructure and services.Advantage skillsExperience working with Kubernetes and Spark stacks together.Experience working with the Apache Druid / Hadoop stack in operating environments with a large amount of data.Familiarity with ETL tools such as Apache Drill and Presto.